1. Field of the Invention
The present invention relates to automated test assembly.
2. Description of the Related Art
The following describes common terms used in test development. “Item” is a term designating a question that can be placed on a test. An item has Item Response Theory (“IRT”) parameters (discrimination, difficulty, guessing), a cognitive skill subtype and an associated passage. A passage (or stimulus) is used to introduce a group of items. A passage has the following properties: topic, social group orientation, number of words, type, and related items. A section is a collection of items administered together without break. Every section must satisfy stated constraints that can vary from section to section. A test is a combination of sections satisfying the stated test constraints. In some cases, passages are used. In these cases, each item has one related passage and each passage has one or more related items. As referred to herein, a “sub-pool” can be a test or section or any subset thereof.
Thus, a test is composed of items (test questions) and passages (stimulus material to which the items refer). A test is scored based on an examinee's responses to the items. If there is a one-to-one correspondence between an item and its passage (or if there is no passage), the item is called discrete. If more than one item is associated with a passage, the item is called set based. A database of items, passages and their associated characteristics is called an item bank or item pool. Usually, the type of a section corresponds to the type of its items. If two tests (or sections) have one or more passages (or items) in common, they are called overlapping (also referred to herein as intersecting); otherwise, they are non-overlapping (also referred to herein as disjoint).
The last two decades has seen a wide spread usage of automated test assembly at testing agencies. Most practical test assembly problems are NP-Complete. Thus, no polynomial algorithm exists for their solution and a search procedure must be used for large problems. This does not mean that the assembly of a single linear test is difficult. Most test assembly situations do not require the optimization of an objective function. Test specifications are defined and any combination of items meeting the specifications yields an acceptable test. A typical item pool would give rise to a large number of ways to combine items to make a test. Heuristics methods for test assembly are described in the prior art. In certain prior art methods, a combination of network flow and Lagrangian relaxation for test assembly is utilized. In other prior art methods, the use of a more general mixed integer programming (MIP) code was proposed. The MIP approach is now considered a common technique to assemble tests, although it does not support non-linear constraints.
Adaptive stochastic search, including simulated annealing, genetic algorithms, tabu search, and Monte Carlo methods (random search, Markov chain simulation), are being successfully used for various practical global optimization problems. This success is due to easy implementation and adaptation to the complex problems. The present invention presents a new test assembler exploiting stochastic methods and supporting linear and non-linear constraints.
One issue relating to test assembly involves the problem of identifying multiple non-overlapping tests. The problem of finding the maximum number of non-overlapping tests is referred to herein as the extraction problem. Taking into account the cost of development and maintenance of each item in a pool, the solution of the extraction problem has great value for a testing agency. The conventional approach is to assemble tests sequentially while removing from the pool any previously used items and passages. This technique can not guarantee the optimal solution to the extraction problem because removal of items can block further assembly of non-overlapping tests. The present invention presents a new approach that does not have this disadvantage.